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Abstract. We consider long term average or 'ergodic' optimal control poblems with a special 
structure: Control is exerted in all directions and the control costs are proportional to the 
square of the norm of the control field with respect to the metric induced by the noise. The long 
term stochastic dynamics on the manifold will be completely characterized by the long term 
density p and the long term current density J. As such, control problems may be reformulated 
as variational problems over p and J. We discuss several optimization problems: the problem 
in which both p and J are varied freely, the problem in which p is fixed and the one in which 
J is fixed. These problems lead to different kinds of operator problems: linear PDEs in the 
first two cases and a nonlinear PDE in the latter case. These results are obtained through 
through variational principle using infinite dimensional Lagrange multipliers. In the case where 
the initial dynamics are reversible we obtain the result that the optimally controlled diffusion 
is also symmetrizable. The particular case of constraining the dynamics to be reversible of 
the optimally controlled process leads to a linear eigenvalue problem for the square root of the 
density process. 
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f. Introduction 

In this paper we discuss stochastic, long term average optimal, or 'ergodic' control problems 
on compact orientable manifolds. The theory about ergodic control theory in continuous spaces 
has been developed relatively recently; see works by Borkar and Gosh (e.g. [5]) and the recent 
monograph pQ. To our knowledge no literature is available about this topic in the setting of 
compact manifolds. 

We concentrate on a special case of the control problem, in which control is exerted in all 
directions and where the control costs are proportional to the square of the norm of the control 
field with respect to the metric induced by the noise, as discussed in Section [3] As such, our 
emphasis does not lie on the solution of applied control problems. This setting may however prove 
relevant for obtaining results in large deviations theory; see e.g. where the connection is 

made between control problems and large deviations theory. The 'squared control cost' is further 
motivated by recent results we obtained on stochastic optimal control for finite time horizon 
problems, with relative entropy determining control cost 0J. This particular setting typically 
leads to linearized systems [12]. In the ergodic setting it leads typically to operator eigenvalue 
problems, see e.g. [18] for the diffusion case and [20] for the Markov chain setting. 

On a compact manifold, a few phenomena play a special role. The main advantage of this 
setting is that transient behaviour cannot occur. Therefore, an invariant measure is necessarily 
unique and ergodicity follows immediately. The long term stochastic dynamics on the manifold 
will be completely characterized by the long term density p and the long term current density J 
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(see Section [3]). As such, control problems may be reformulated as variational problems over p and 
J. In the optimization problem, the density p is paired with the scalar cost or potential function 
V, and the current density J is paired with a vector potential or gauge field A to obtain the cost 
function. In Section [4] we discuss how to understand the notion of flux as a particular example of 
choosing a gauge field A. 

We then discuss several optimization problems: the problem in which both p and J are varied 
freely (Section [5J the problem in which p is fixed (Section [5]) and the one in which J is fixed 
(Section [7]). These problems lead to different kinds of operator problems: linear PDEs in the first 
two cases and a nonlinear PDE in the latter case. These results are obtained through through 
variational principle using infinite dimensional Lagrange multipliers. This analysis is performed 
rigourously in Section [5] 

In the case where the initial dynamics are reversible, or in other words, in case the diffusion is 
symmetrizable, we obtain the result that the optimally controlled diffusion is also symmetrizable 
fSection l5.4p . The particular case of insisting J = coincides with demanding reversible dynamics 
of the optimally controlled process. Interestingly, this optimization problem leads to a linear 
eigenvalue problem for the square root of the density process, just as we see in quantum mechanics 
(but note that our setting is entirely classical). We conclude this paper with a short discussion 
(Section [HD - 

This paper is written for a mathematical audience. The reader interested in the statistical 
physics interpretation of this material is referred to our related publication [6] . 

2. Problem setting 

We will phrase our setting in terms of diffusion processes on manifolds, in the language of [HI 
Chapter V]. By smooth we always mean infinitely often differentiable. M will always denote a 
smooth compact orientable m-dimensional manifold. Let C°°(M) denote the space of smooth 
functions from M into R, let X(M) denote the space of smooth vectorfields on M, and let hP(M) 
denote the space of smooth differential forms of order p on M, for p = 0, 1, . . . , m. 

Let (fi, F, (.Ft)) IP) denote a filtered probability space on which is defined a d-dimensional stan- 
dard Brownian motion. Consider a stochastic process X defined on M by the SDE, given in local 
coordinates by 

(1) dX\ = ff y (**)/,-(*, X t ) dt + o-i(X t ) o dB?, 

where, for a = 1, . . . , d, a a € X(M), g^ := J2 a "'a^a) f° r iji = 1> • • • , m, is a symmetric positive 
semidcfinitc bilinear tensorfield on M, f is a differential 1-form on M, denoting force. For any 
initial condition xq £ M, let X x ° denote the unique solution to ([1]) The notation odB a indicates 
that we take Stratonovich integrals with respect to the Brownian motion. One can think of / as 
a force field, resulting from a potential, some external influence, or a combination of both. 
We will always assume the following hypothesis. 

Hypothesis 2.1 (Ellipticity). g is positive definite on M, 

Under this assumption, g defines a Riemannian metric on M and we will use this as the metric 
of choice without further notice. This Riemannian metric induces a local inner product (•, •) and 
corresponding norm || • || on tensors of arbitrary covariant and contravariant orders. 

The SDE {TJ) is referred to as the uncontrolled dynamics. These dynamics may be altered by 
exterting a 'force' or 'control' u in the following way, 

d 

(2) dX\ = g ij (X(t)) {f 3 (X t ) + Uj (X t )} dt+J2 <( x t) ° dB", 

a=l 

where u € A 1 (M). For any xq G M and u £ A 1 (M), the unique solution to ([2]) will be denoted by 
X x °- U . The SDE ((2|) is referred to as the controlled dynamics. 
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Consider the random functional C : ft x M x A 1 (M) — > R denoting pathwise long term average 
cost, 



(3) C(u,x ,u) := limsupi 



T 1 rT 



V(Xf°' u ) + —\\u(X* - u )\\ 2 ds+ I A(Xf' u ) o dX*°' u 
2A 



where A > 0, V £ C°°(M) is a potential or s£a£e dependent cost function, ||it(-)l| 2 represents the 
(instantaneous) control cost corresponding to a control field u £ A (Af), and A G A x (Af). The 
final term in ([2]) may represent a /?icr, as explained in Section @] The differential form A is often 
called a gauge field in physics. 

Remark 2.2. The 'limsup' in <j3j) is used to avoid any discussion at this point about the existence 
of the limit. Instead of the pathwise formulation in ([3]) we could alternatively consider the weaker 
average formulation, in which case the cost function would be the long term average of the expec- 
tation value E 10 '" of the integrand in ©. We will see in Section [3] that the limit of ^ exists (and 
not just the 'limsup'). Furthermore this limit will turn out to be equal to a deterministic quantity, 
so that the pathwise formulation and the average formulation may be considered equivalent. 

We will consider the following problem, along with some variations which we discuss in Sec- 
tions [5] and [7] 

Problem 2.3. For every xq £ M, find a differential 1-form u £ A 1 (M) such that 

C(xq,u) = inf C(xq,u), almost surely. 

ueA 1 (M) 



3. ERGODIC REFORMULATION OF THE OPTIMIZATION PROBLEM 

In this section we will derive two equivalent formulations of Problcm l2.3l These reformulations, 
Problem 13.81 and Problem 13 . 1 31 below, are better suited to the analysis in the remaining sections. 
Also some notation will be established that will be used throughout this paper. 

Let ft x = C([0, oo); M) denote the space of sample paths of solutions to ©. We equip Vl x with 
the er-algebra T x and filtration (T x )t>o generated by the cylinder sets of X. Furthermore let 
probability measures P x °> u on fl x be defined as the law of X Xo < u 7 for all x £ M and u £ A 1 (M). 
Note that for all u £ A 1 (M) the collection of probability measures P ' u defines a Markov process 
on tt x . 

For the moment let u £ A 1 (M) be fixed. It will be convenient to use the shorthand notation 

bi(x)=g»(x)[f j (x) + u j (x)}. 

Recall that associated with the vectorfields <r a , a = l,...,d, there exist first order differential 
operators also denoted by a a : C°°(M) — > C°°(M) defined by 

a a f{x) = a i a {x)d i f{x), x £ M, 

for / £ C oc (M), where di = denotes partial differentation with respect to x 1 . Similarly b u 
defines a first order differential operator also denoted by b u . By [HJ Theorem V.1.2], the Markov 
generator corresponding to © is given by 

d 

L u (f)(x) = | ^ o- a <7 a (j>(x) + b u (f)(x). 

a=l 

On A P (M) an inner product is defined by (a, f3) ap(M) — Jm^"^) ^• c ' wnere dx denotes the 
volume form corresponding to g. The inner product (•, -)a°(m) is a l so denoted by (■, ■) l 2 (m)- Let 
L 2 (M,g) — L 2 (M) denote the usual Hilbert space obtained by completing C°°(M) with respect 
to the L 2 (M)-inner product. 

Lemma 3.1. L u may be written as 



L«$ = |A$ + (u + /,d$), $eC 2 (M), 



4 JORIS BIERKENS, VLADIMIR Y. CHERNYAK, MICHAEL CHERTKOV, AND HILBERT J. KAPPEN 

where A the Laplace- Beltrami operator, /, := fi+hgik (V ' a a o~ a ) k ' , anil's/ is the covariant derivative, 
corresponding to the Levi-Civita connection of the Riemannian metric g. The adjoint of L u with 
respect to the L 2 (M) inner product is given by 



L*J> = ±A* + <5 (*(/ + «) J , *eC 2 (M). 

where 5 : A P (M) —> A p (M) is the L 2 (M,g) adjoint of the exterior derivative operator d, i.e. 

j (da,P)dx=[ aS/3 dx, a € C°°(M), (3 € A 1 (M), 
Jm Jm 

where dx denotes the measure induced by the volume form on M . 

Proof. The Laplace-Beltrami operator may be expressed as (see [131 p- 285, eqn. (4.32)]) 

(4) Ao g'Y l ;n,o. 

where 1^ denote the Christoffel symbols corresponding to the Levi-Civita connection. Using this 
expression, we compute 

E = E < d * HW) = E (<WW + < &<) (d^)) = E (JI' J <>.<>/:' + < (diPi) (M) 

a a a a 

= + g ij T^d k( f> + E < d ^ = ^ + E ( v ^«) <t>, 

a a 

where the last equality is a result of the definition of the Levi-Civita connection and the corre- 
sponding Christoffel symbols. The expression for L* is immediate from its definition. □ 

In the remainder of this work, we will assume all advection terms are absorbed in the force 
field /, and thus may omit the tilde in /. This can alternatively be interpreted as assuming 
Y] Vo-^CTq = 0. This is further equivalent to the following hypothesis. 

Hypothesis 3.2. The generator corresponding to X equals L u — ^A + b u . 

This assumption is justified by the above lemma and the well-known fact that Markov generator 
L u determines the law of the diffusion X uniquely. 

Lemma 3.3. Let xq £ M . The expectation of the trajectory of X over the gauge field may be 
expressed as 

E x ,u f A{Xt) dXt = W o,u r ^ f + u}- \8A] {X t ) dt, 
Jo Jo 

or in local coordinates, 

W0 ,u ( T A{Xt )odX t =E*- U C g ik A i (fk + u k )(X t ) + l^=d j (V\9\9 ij Ai) (X t ) dt. 
Jo Jo V If I ' 

Proof. Using the usual transformation rule between ltd and Stratonovich integrals |14[ Equation 
(1.4), p. 250], we may write 

d 

(5) dX\ = t u (X t ) dt + J2 <(Xt)dB a (t), 

a=l 

where b u (x) is given by 



a=l 

By the definition of the Stratonovich integral, Z o dY = Z dY + \d\Z, Y] for semimartingales Y 
and Z Q31 Equation (1.10), p. 100], with Z dY denoting the ltd integral. Therefore 

Ai{X t ) o dX\ = Ai(X t ) dX\ + \d[Ai{X t ),Xi] = A{X t ) dX\ + ±d 3 Mx t )d[x{ , XI] 



A l {X t 



t u {X t ) dt + J2ai(X t ) dB a (t) 



^Mx^o-UXtyiiXt) dt. 
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Integrating over t and taking expectations gives 



f A(X s )odX s 
Jo 

E Xo,u f {A t (X t ) [bi(X t ) + i (dMX t )) a k a (X t )] + ^A(X t )ai(X t )a l a (X t )} dt 



W o,u / { g ik A .( fk + Uk )( Xt ) + ^V^AiiXt)} dt. 



In the last expression we recognize the divergence of the vectorfield <? IJ Ai, resulting in the stated 
expression. □ 

We recall the notion of an invariant probability distribution. Let B(M) denote the Borel a- 
algebra on M. 

Definition 3.4. A probability measure p, u on M is called an invariant probability distribution 
for O, if 



f F x ' u (X t £ B)p u {dx) = fx u (B), B £ B{M). 
Jm 



The following result on invariant measures for nondegenerate diffusions |T?l Proposition V.4.5] 
is essential for our purposes. 

Proposition 3.5 (Existence and uniqueness of invariant probability measure). Suppose Hypoth- 
esis \2.1\ is satisfied. Corresponding to any u £ A (M) there exists a unique invariant probability 
measure p u on M corresponding to the diffusion on M defined by ([2]) . Moreover, fi u (dx) is given 
as p u {x) dx, and p u g C°°(M) is a solution of 

(6) L* u p = 0. 

Furthermore p u > on M . 

We will refer to (JSJ as the Fokker-Planck equation, in agreement with the physics nomenclature. 
In the remainder of this work let [i u and p u as defined by Proposition [23J 

In the physics literature, the empirical density and empirical current density are defined re- 
spectively as (see [7]): 

p t (x,uj) = - / S(x - X a (w)) ds, J t (x,u) = - / X s S(x - X 3 (u)) ds. 
t Jo 4 Jo 

Here (and only here) 5 denotes the Dirac delta function. These fields, having a clear heuristic 
meaning, will be very relevant in the remainder of this work and we will make these precise from 
a mathematical point of view. 

Let Bb{M) denote the set of bounded Borel- measurable functions on M. We will work with 
the set of empirical average measures (vt(dx,Lj)) t>0 on B(M) x Q x , defined by 



1 /■' 

B):=- 1 B (X S ) ds, t>0,BeB(M), 
t Jo 



(7) M 

where 1^ denotes the indicator function of the set B. Note that the measure p,t(B) := J * 1b(X s ) ds 
is known as the local time of X. Our primary interest is in the infinite time horizon limit. 

Proposition 3.6. For all u € A 1 (M),(p £ L 2 (M,p u ) (in particular for <p £ Bb(M)) and xq £ M, 

(8) lim / <p dvt = / <p dp u , p x "< u -almost surely. 
t ^°° Jm Jm 

Proof. For u £ A 1 (M), we define a stationary probability measure V u on £l x by 

P U (G) = f F X °> U (G) p u (dy), G £ T. 
Jm 
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For ip £ L 2 (M 7 p u ) we then have, by the ergodic theorem, see e.g. [5J Theorem 3.3.1], that 
lim t _ i . 00 J M ip dv t — j M <p dp, u , P u -almost surely. Since p u > on M, this implies that lim t _). 00 J M ip dv t = 
J <p dfi u , P K °'"-almost surely for /Lt u -almost all x$ g M. By smooth dependence of the trajectories 
of X on the initial condition the result extends to all xq £ M. □ 

Note that limt_>oo v t is deterministic and does not depend on the choice of the initial condition 

X e M. 

Corollary 3.7. For x £ M, 

(9) lim i / A(X t )odX t = [ {{A,f + u)-UA\p u dx, P x "^-a.s. 

T^oo i J J M 



Proof. This is an immediate corollary of Lemma 13.31 Proposition 13.51 and Proposition 13.61 □ 

By the above results we may rephrase Problem 12.31 as follows. 
Problem 3.8. Minimize 

(10) C{ P ,u) := |v + i-|| w || 2 + (A, f + u}- \5A^ p dx. 

with respect to (p, u) £ C°°(M) x A 1 (M) subject to the constraints L* u p = and J p dx = 1. 

Remark 3.9. The gauge field A may be completely removed from the problem by redefining / and 
V to be 

(11) f = f-XA, and V = V - ±A||A|| 2 - \8A. 
It may be checked that Problem 13.81 is equivalent to the minimization of 

C(p,u)= [ [v+ l-\\u\\ 2 } p dx, 



2A 1 



with respect to p and u, subject to ^Ap + 6((f + u)p) — 0. The control u solving Problem 
for nonzero A may be retrieved by setting u = u — XA. Using this observation would simplify 
the derivation of the results in subsequent sections, but the process of reintroducing a nonzero 
gaugefield A in the results would lead to unnecessary confusion, so we will continue to work with 
a nonzero gaugefield A. 

Note that X s is not defined, a.s., so our mathematical analogue of the empirical current density 
requires more care. In Appendix 1X1 we derive the differential 1-form J u £ A 1 (M) denoting current 
density, as 

(12) J u = -\dp u + p u {f + u). 

Note that J u is divergence free: 5J U — L* u p u — 0. Furthermore a control u may be expressed 
in terms of the corresponding J u and p u as 

(13) u = -f + — {J u + \dp u ) • 

Pu 

Lemma 3.10. Suppose u, p u and J u are related by (TT2"]) . Then SJ U = if and only if L* u p u = 0. 
i.e. p u is an invariant measure. 

Proof. This follows immediately from noting that SJ U — L* u p u . □ 

Recall Lemma T3.31 where the expectation of the gauge field over the trajectory was expressed 
as an expectation over a Lebesgue integral. For the long term average of the gauge field this leads 
to the following result. 

Lemma 3.11. For u £ A 1 (M) and x a £ M, 

(14) lim 1 / A(X t )odXt= ( {A,J u )dx, P*°>«-a.s. 

T^oo 1 J J M 
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Proof. From Corollary 13. 71 we have ©■ By (fl~3")l . this is equal to 

\Ja, y (Ju + ypu)j - ISAj Pu dx = (A, (J u + \d Pu )) - ±{A,dp u ) dx = J {A, J u ) dx. 



□ 



Because of the above observations, instead of varying p and u in the optimization problem [ 
we may as well vary p £ C°° and J € A 1 (M), while enforcing S J — (equivalent to the Fokker- 
Planck equation for p by Lemma |3.10[) and J M p dx = 1. Because of (fT3")) the control u is 
then determined explicitly. Combining (|1 3[) and (|14j) . we may alternatively express the cost 
functional (|10p as a function of p and J, namely 



(15) C{p,J) = 




-f + - ( J + Idp) 



p + (A, J) ^ cfe. 



Remark 3.12. Strictly speaking the use of C for different cost functionals is an abuse of notation; 
we trust this will not lead to confusion. 

Problem (|2.3|) can thus be rephrased as the following problem: 

Problem 3.13. Minimize C(p,J) with respect to p £ C°°(M) and J € A 1 (M), subject to the 
constraints S J = and J M p dx = 1, where C(,o, J) is given by (|15[) . 

4. Flux 

In this section we will give a natural interpretation of the term J M (A,J) dx, namely as the 
flux of J through a cross-section a, or equivalently, the long term average intersection index of the 
stochastic process (X(t)) t >o with a cross-section. The section motivates the gauge field A in the 
cost function. The remainder of this paper does not refer to this section. For background reading 
in differential geometry, see [SJ [21] . See also [6) [7] where the ideas below are described in more 
detail. 

Let M be a compact, oriented, Riemannian manifold of dimension m. Recall the notion of 
a singular p-chain in M (with real coefficients) as a finite linear combination c = ^ciiai of 
smooth p-simplices <r,; in M where the are real numbers. Let S P (M;M) denote the real vector 
space of singular p-chains in M. On S P (M;M.), p 6 Z, p > 0, are defined boundary operators 
dp : S P (M;M) — > 5 p _i(M;R). The p-th singular homology group of M with real coefficients is 
defined by 

H p (M;R) = kerdp/imdp +1 . 

Elements of ker9 p are called p-cycles, and elements of im<9 p+ i are called p-boundaries. The 
deRham cohomology classes H^ cKham (M;R) are defined, for < p < m = dim(M) as 

#deRham( M ) = kerd p / imd p _i, 

where d p : A P (M) — > A P+1 (M) denotes exterior differentiation. 

Let a be a p-cycle. The functional p a £ A P (M) — > J f3 £ R depends, by Stokes' theorem, 
only on the homology class [a] of a, and the deRham cohomology class [f3] of (3. An element 
[a] £ H p (M;M.) may therefore be considered an element of (-HdcRham(-^O)*- N° w let M be 
oriented and of dimension m. The mapping qp : [7] £ H^ eRhiim (M) — > Jm@ A 7 G R is an 
element of {H^ cRham (M))* . Poincare duality states that the mapping [f3] £ H^£ am (M) —> 
IP e (Hl eRha jM))* is an isomorphism for compact M, i.e. B^JM) = (^ eRham (M))*. 
Therefore, for compact, oriented M, we have 

H p (M;R) - (i/ d p cRham (M))* = K e ^ m (M). 

Since an equivalence class in H^^ am {M) has a unique harmonic representative, we conclude that 
for a p-cycle a, there exists a unique harmonic r a £ A m ~ p (M) such that 

P = P«C8) = / r a A [3, [f3] £ H p cRham (M). 

JM 
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In particular, for p = m — 1, we may interpret J -kj (with ★ the Hodge star operator) as the flux 
of J through a. This quantity may further be interpreted as the long term average intersection 
index of the stochastic trajectory (X(t))t>o with respect to a, i.e. the long term average of the 
number of intersections (with ±1 signs depending on the direction); see e.g. |T3l Section 0.4]. 
Specializing the above result to this situation, we obtain the following proposition. 

Proposition 4.1. For a given (m — l)-cycle a, there exists a unique harmonic A £ A 1 (M), 
which depends only on the singular homology class [a] £ JT m _i(M;M) of a, such that f a *J = 
J M {A, J) dx for all J £ A 1 (M) satisfying 6 J = 0. 

Example 4.2 (S 1 ). The divergence free 1-forms J on S 1 are constant, say J — Jq d8 for some 
Jo £ R. A 0-cycle a of S 1 consists of a collection of points 6\, . . . , Ok C [0, 2tt] with multiplicities 
ai,...,afe. The flux of J through a is then simply given by J2i=i a iJ(®i) = «i Jo- By 

defining a differential form A = Aq d8, with constant component Aq := a i^> we nn( i that 

/ (A, J) dO = { AqJ d<9 = V ai J = [ *J. 

JS 1 JS 1 i Ja 

We see that this choice of A is the constant (and therefore harmonic) representative in H^^^^S 1 ) 
corresponding to [a] € 7Jo(S' 1 ;IR). 

5. Unconstrained optimization - the HJB equation 

In this section we will find necessary conditions for a solution of Problem 13.81 or equivalently 
Problcm l3.13l In fact, for technical reasons we will work with the the formulation in terms of p and 
J, i.e. Problem 13. 131 The main reason for this is the simplicity (in particular, the linearity) of the 
constraint 5 J — 0. This should be compared to the equivalent constraint ^Ap + S(p(f + uj) = 0, 
which is nonlinear as a function in (p, u). 

The approach to Problem 13. 81 or Problem 13. 131 is to use the method of Lagrange multipliers to 
enforce the constraints. Since the constraint SJ(x) = needs to be enforced for all x £ M, the 
corresponding Lagrange multiplier is an element of a function space. A purely formal derivation 
of the necessary conditions using Lagrange multipliers is straightforward, but we wanted to be 
more rigourous in the derivation of the necessary conditions. In Sections |5] and [71 we will be less 
rigourous in the derivations, in the comforting knowledge that we can use the machinery outlined 
in the current section. 

5.1. Abstract optimization. We will relax Problem 13.131 to an optimization problem over 
Sobolev spaces. In particular, we will rephrase it as the following abstract optimization prob- 
lem. Let X and Z be Banach spaces and let U be an open set in X. Let C : U C X — s- R and 
H : U C X ->■ Z. 

Problem 5.1. Minimize C(x) over U subject to the constraint H(x) = 0. 

The Frechet derivative |15) of a mapping T : D a X — > Y in x £ D will be denoted by 
T'(x) £ L(X;Y). We will need the following notion. 

Definition 5.2 (Regular point). Let T be a continuously Frechet differentiable function from an 
open set D in a Banach space X into a Banach space Y . If xq £ D is such that T'(xo) maps X 
onto Y, then the point xq is said to be a regular point of the transformation T. 

For the Problem lS. li the following necessary condition holds for a local extremum [T5] Theorem 
9.3.1]. 

Proposition 5.3 (Lagrange multiplier necessary conditions). Suppose C and % are continuously 
Frechet differentiable onU. If C has a local extremum under the constraint "H(x) = at the regular 
point xq £ U, then there exists an element Zg £ Z* such that 

f'(x ) + (H'(x ),zZ)=0. 

Here (•, •) denotes the pairing between Z and Z* . 



ERGODIC OPTIMIZATION ON MANIFOLDS 



9 



5.2. Sobolev spaces. We will define Sobolev spaces of differential forms as follows. For k G 
N U {0} and p G {0, . . . , n}, the space W k,p (M) consists of all differential forms of order p which 
are in L 2 (M) together with their covariant derivatives up to order k. (Note the special meaning 
of the second index p, in contrast with the more common definition.) The norm 



\M\L--= I Eiiv^ii 2 ^ 

J M , n 



gives Hilbert space structure to W k ' p (M). We will write W k (M) := W kfl {M). 

Note that the Laplace-Beltrami operator A = — (5d+dS) is a bounded mapping A : W ' P (M) —> 
W k - 2 ' P (M). Also S : W k ' p (M) W k ^ 1 ' p ^ 1 (M), for p > 1, and d : W k > p {M) W fc - 1 ' p+1 (M), 
for p < n, are bounded linear mappings. 

Recall Sobolev's Lemma [El Proposition 4.3.3]. 

Lemma 5.4 (Sobolev embedding). Suppose lu G H k (M) and suppose m G N U {0} satisfies 
k > n/2 + m. Then lu G C m (M). 

5.3. Obtaining necessary conditions for the optimization problem. We let k G N such 
that 

(16) k > n/2. 

For this k, we define 

X := W k+1 (M) x W k ' 1 (M), 

U :=V x W k > x {M), where V := {p G W k+1 (M) : p > on M) , and 

Z:=Z!XR, where Z\ := {ijj G W k ~ 1 (M) : = 8(3 for some (3 G W k,1 (M)} . 

By the condition (fl6|) on k, we have by the Sobolev Lemma that (p, J) £ X satisfies p G C(M). 
In particular, the condition p > defines an open subset U G X. 

Lemma 5.5. Suppose (|16l) holds. The mapping C{p, J), as given by (I15|) . may 6e continuously 
extended to a mapping C : W — > R. Moreover, the mapping C is continuously differ entiable on IA 
with Frechet derivative C'(p, J) G L(X; R) given for (p, J) G U by 

(C, GO * / m {v + - JL||J + ^ll a + ^ (-/ + ~ (J + ¥p)) } Cd* 

(K _/+ p (j+ ^ p) ) +A,G ) 



dx. 

Im 

Proof. We compute C'(p,u) to be the linear functional on X given by 

2 



(C,G)^ / 

J A 




C+±(-f+±(j+idp),~{j+idp)t + ±dC 



This is after rearranging, and partial integration of the term containing d£, equal to the stated ex- 
pression. The derivative is a bounded functional on X by the uniform boundedness of V,l/p, dp, f, J 
and A. □ 



We define the constraint mapping T~L : U — > Z as 

H(p, J) := (sj, j pdx-lj, 
The following lemma is now immediate. 



(J, It) G 



in 
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Lemma 5.6. The mapping T-L is continuously differ entiable on X , with Frechet derivative %' '(p, J) G 
L(X; Z) given for (p, J) G X by 



(CG)h- ISG, J (t,v)€X. 

Every (p. J) G X is regular for H, thanks to our choice of the function space Z: 
Lemma 5.7. Any (p, J) G X is a regular point ofH. 

Proof. Let k) € Z = Z 1 X R. In particular there exists a /3 G W fc,1 (M) such that * = J/3. We 
may pick G — (3, and £ a constant function such that J M C, dx = n. Then "W(/9, J)(C, G) = k), 
showing that H'(p, J) is onto. □ 



In order to apply Proposition 15.31 in a useful manner, we need to give interpretation to Z*, 
and in particular to Z\. First let us recall that the spaces (H S (M))*, for s G N U {0} may be 
canonically identified through the L 2 (M)-inner product with spaces of distributions, denoted by 
W~ S (M) [El Proposition 4.3.2]. In other words, if z G (H S (M))*, then there exists a distribution 
$ G W~ S (M) such that z(*) = J M dx. Now in case $ 6 Zj, i.e. = S/3 for some 
P G W^M), then z(*) = J M <f>5j3 dx = J M (d<S>,/3) dx for some $ G W~^ k ~^ (M). Therefore 
the choice of $ G W~( k ~ x \M) representing z G Z\ is fixed up to the addition by a closed form: 
if d 1 = for 7 G W-^-^Af), then / M ($ + 7 )5/3 = J M $<5£ dx. We summarize this in the 
following lemma. 

Lemma 5.8. Z\ ^ W^ k ^{M)j {7 G W~^ t ~ x ^{M) : d 7 = 0}. 

We may now apply Proposition 15 . 31 to obtain the following preliminary result. 



Lemma 5.9 (Ergodic Hamilton- Jacobi-Bellman equation). Suppose (p, J) GW is a local extremum 
of C, defined by (|15[) . under the constraint that TL(p, J) — 0. Then there exists $ G C (M) and 
7 G K such that 

V + {f,A + d<$>) - ±A||/1 + tM>|| 2 - ±5{A + d<S>) +7 = 0, and 
(17) J = -\dp + pf- \p{A + d$), 

where the first equality holds in weak sense, i.e. in W (M ). The corresponding control field u is 
continuous and given by 

u = -\(A + d$). 

Proof. Let (p, J) as specified. By Proposition l5.31 there exists an element ($, 7) G VT - ^ -1 ' (M)xl 
such that the following equations hold. 

V + k W ' 2A7 11 J + ^ Pl ' 2 + Y\ 5 ( _/ + ~p (J+ ^ dp) ) + 7 = °' and 

j y-f + - p {J + \dp)^j + A + d<S> = 0. 

Here $ is an arbitrary representative of some equivalence class in W'^'^iM)/ {7 G W^ k ~^ (M) : dj 
as motivated by Lemma 15.81 Substituting the second equation into the first, and making some 
rearrangements, gives the sytem (fl~7| . Then $ G C 1 (M) as a result of the equation for J in (|I7I) 
and the continuity of J, p and dp. The expression for u is an immediate result of (|13p. □ 



Theorem 5.10 (Linearized HJB equation). Suppose (p, J) G IA is a local extremum of C, defined 
by (|15|) . under the constraint that H(p, J) — 0. Then there exists a ip G C°°{M), tp > on M, 
and smc/i i/iai 

-ffi/' — Wip = pip, where 
(18) Hip := L uo ip, u = -XA, and W = XV + \{f , A) - \\ 2 \\A\\ 2 - \\8A. 
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Furthermore p G C°°(M), J G A 1 (M) and the control field u, related to (p,J) as in (1131) . is in 
A (M) as well, and 

u = —\A + d (In ip) , J = - \dp + pf - XpA + p d (In ip) . 
Proof. Let $ and 7 as in Lemma I53l and let ip := exp(— A$). We compute 

d$ = --"-# and 5d<P = -^-6 dip - I 2 . 
Xip \ip \ip z 

Inserting into (flTj) . and multiplication by A^ makes the \\dij)\\ -terms cancel and results in the 
equation 

XipV + Xip(f,A) - (f,dip) - ±\ 2 ip\\A\\ 2 + X(A,dip) - ±XipSA + ±8diP + \jiP = 0, 

or equivalently (|18|) . with p = A7. 

Regularity. Note that, since ip G C^Af), we have dip G C(M) C L 2 (M), so that V € iJ x (M). 
Note that the first equation of (fT5|) may be rewritten into \A-ip = with ¥' = —(/ — XA,dip) + 
Wtp + pip € C(M) C L 2 (M). By elliptic regularity ([191 Proposition 5.1.6]), it follows that ^ G 
H 2 (M). In particular, 95 € H 1 (M). We may iterate this bootstrapping argument to conclude that 
?/> G H S (M) for any sgN, and conclude from Sobolev embedding (Lemma 15 .4[) that ip € C°°(M). 

By (fT3|). the corresponding control field is given by u = —AA + -kdip € A X (M). By Proposi- 
tion^ p G C°°(Af) and by the equation for J in jJ7|, J G A X (M). □ 

Since the problem considered in Theorem l5.10l is a relaxed version of Problem l3.13[ and smooth- 
ness of J and p is established in the relaxed case, we immediately have the following corollary. 

Corollary 5.11. Suppose (p,u) is a solution of Problem \3.8\. or equivalently that {p, J) a solution 
of Problem \3.13i Then the results of Theorem \5.10\ hold. 

Remark 5.12. This result should be compared to [5], in which a variational principle is derived 
for the maximal eigenvalue of an operator L satisfying a maximum principle. Our results give an 
alternative characterization (as the solution of a control problem) of the largest eigenvalue in the 
case of an elliptic differential operator L = L Uo . 

Remark 5.13. An alternative way of deriving the HJB equation is by using the method of vanishing 
discount, see e.g. [U Chapter 3]. 

5.4. Symmetrizable solution. In a special case we can represent the optimally controlled in- 
variant measure in terms of ip and the uncontrolled invariant measure. For this, we recall the 
notion of a symmetrizable diffusion [141 Section V.4]. Other equivalent terminology is that the 
Markov process reversible or that the invariant measure satisfies detailed balance. 

Definition 5.14. Let (T(t)) t >o denote the transition semigroup of a diffusion on M. This diffusion 
is said to be symmetrizable if there exists a Borel measure v(dx) such that 

f (T(t)f)(x)g(x) vdx= f f(x)(T(t)g)(x) v dx for all f,g€ C(M),t > 0. 

JM JM 

In case a diffusion is symmetrizable with respect to a measure is, this measure is an invariant 
measure for the diffusion. 

The following results hold for any control field u G A 1 (Af). 

Lemma 5.15. Let X denote a diffusion with generator given by Lh = + (/ + u,dh), h G 
C 2 (M). The following are equivalent. 

(i) X is symmetrizable, with invariant density p u = exp(-U) for some U G C°°{M); 

(ii) f + u = -\dU for some U G C°° ; 

(iii) The long term current density J u , given by (|12p . vanishes. 



Proof. The equivalence of (i) and (ii) is well known, see e.g. [141 Theorem V.4. 6]. The equivalence 
of (ii) and (iii) is then immediate from ([12"]) . □ 
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Proposition 5.16. Let /?o an d Jo — ~\dp§ + Pof denote the density and current corresponding 
to the uncontrolled dynamics ([1} . The following are equivalent. 

(i) The diffusion corresponding to the optimal control u is symmetrizable, with density p = 
ip 2 po, where ip is as in Theorem \5.10\ (normalized such that J M ip 2 po dx = 1); 

(ii) J = Xp A; 

(iii) / = -\dU + XA, for some U G C°°(M). 

In particular, if the uncontrolled diffusion is symmetrizable and A — 0, then the controlled 
difussion is symmetrizable and the density admits the expression given under (i). 

Proof. Setting p u = ip 2 po, we have 

Ju = - \dp u +p u (f-XA- = -\i> 2 dp - poipdip + i/j 2 p (f - XA + ^dip^j 

= -yj 2 d Po + ip 2 Po (f - XA) = tA 2 (J - X Pa A), 

which establishes the equivalence of (i) and (ii). Representing the density po by exp(— U) and 
using (|12p with u = gives the equivalence between (ii) and (iii). □ 



5.5. Gauge invariance. For a special choice of A, the solution of Problem 13. 131 may be related 
to the solution corresponding to A — in a simple way. 

Proposition 5.17. Let A a G A 1 (M). For <p G C°°(M) and A = A + dip let (p v , J v ) denote the 
solution of Problem \3.13\ with corresponding solutions ip^ and u v of (|18[) . Then, for tp G C oc (M), 

(19) p v = p, u v = u, ip v = exp(Xip)ip, and J v = J, 

where p, J, tp and u denote the solution of Problem \3.13\ and (|18j) corrsponding to A = Aq . 

Proof. This is a matter of straightforward computation. □ 

In other words, the solution of Problem l3. 131 depends (essentially) only on the equivalence class 
of A, under the equivalence relation A ~ B if and only if A = B + dip for some ip £ C°°(M). 

Remark 5.18. A standard way in physics to obtain gauge invariant differential operators is to 
replace the derivatives with 'long' derivatives. This is illustrated by the observation that the 
operator H of ([T8|) may be expressed, using the Hodge star operator * : A P (M) — > A n ~ p (M), 
p = 0, . . . , n, as 

Hip = | * (d - XAA) * (<f - AAA) tjj + * (/ * (d - AAA)) ip - XVip. 

The operator ip H ► dip — XAA ip is called a 'long' derivative operator. This result is easily obtained 
using the following observations 

■k (a A*f3) = {a, f3), -kd* dcp = A(f>, -kd-ka = —5a, 

for a,P€ A 1 (M) and G C°°{M). 

6. Fixed density 

In this section we consider the problem of fixing the density function p, and finding a force u 
that obtains this density function, at minimum cost. Let p G C°°(M) be fixed, with p > on M 
and J M p dx = 1 . Then for some constant c p we have 



C(p,u) = c p + J j^lMI 2 + (A,u)\ p dx. 



Therefore we will consider the following problem. 

Problem 6.1. Minimize C(u) over A 1 (M), subject to the constraint L* u p — 0, where C(u) is 
defined by 

C ( u )= I {77dM| 2 + (A,u)\pdx. 



M 



2X' 



ERGODIC OPTIMIZATION ON MANIFOLDS 



13 



The corresponding problem in terms of the curent density is the following. As for C(u), terms 
that do not depend on J are eliminated from the cost functional. 

Problem 6.2. Minimize C(J) over A 1 (M), subject to the constraint S J = 0, where C{J) is defined 
by 

(20) c ^-lXK- f+ h dp+xAJ ) + ^-P m2 ) dx - 

By an analogous argument as in Section [5] (but less involved, since we are only optimizing over 
J), we obtain the following result: a relaxed version of Problem 16.21 may be transformed into an 
elliptic PDE. Essentially, this is obtained through variation of the Lagrangian functional 



£(J,$) =C{J)+ [ $SJ dx 
Jm 



Theorem 6.3. Suppose J 6 A (Af) is a local extremum ofC(J) given by (|20[) under the constraint 
that 5J = 0. Then there exists a $ G C°°{M) such that 

(21) A$ + (~ p dp,d$\=-±(±Ap + 5(p(f-\A))). 

For this J is given by 

J = Pf ~\dp- p\(A + d$), 
and the corresponding control field u is given by 

u = -A(A + d$). 

Example 6.4 (Circle). On S 1 every differential 1-form /?, and in particular fJ = f — \A, may be 
written as (3 = — ^ dU + \k d8, where 9 represents the polar coordinate function, U <E C°°(S 1 ) 
and k € R; see e.g. [2U Example 4.14]. Recall that Sf = — div/. Equation (|2T|) then reads 

v> {6 ) + ±*W(0) = - Ye (-pW u 'W + kp ^ 

Based on Remark 16.51 we try a solution of the form $ = — (In p +[/ — </?) . Inserting this into 
the differential equation, we obtain for <f> the equation 

where 7 (0) = $ In p(0) = p> (d) . 

Up to an arbitrary additive constant (which we put to zero), there exists a unique periodic 
solution ip to this differential equation, given by 

fcff/o^pK)" 1 <%-2nfZp(0- 1 dA 
<P(0 ) = — L , ee[0,2ir]. 

Remark 6.5 (Solution in the symmetrizable case). If /— XA = —\dXJ for some U € C°°(M), it may 
be checked that $ = — ^- (In p + c£C/ ) solves (|2T|) . so that the optimal control field u = |d(ln p) — f- 
In other words, the optimal way to obtain a particular density function piif — \A is in 'gradient 
form' is by using a control u so that the resulting force field / + u is again in gradient form, 
/ + u = id(lnp), resulting in reversible dynamics; see also Section [5.41 

Remark 6.6 (Gauge invariance). As in Section l5.5[ it is straightforward to check that a solution 
to Problem 16.21 for A v = Aq + dip is given by 



$ip = $ - <p, p v = p, u v = u, J v = J, 
in terms of the solution ($, p, u, J) corresponding to the gauge field Aq. 
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7. Fixed current density 



In this section we approach the problem of minimizing the average cost, under the constraint 
that J is fixed. In light of the remark just below (fl2|) . it will be necessary to demand that 5 J = 0, 
otherwise we will not be able to obtain a solution. By fj X3[) . we may express u in terms of J and p 
by u = — f + - ( J + \dp). Note that by Lemma fe.lOl the Fokker-Planck equation © is satisfied. 
This leads to the following problem. 

In the remainder of this section let J £ A 1 (M) satisfying 6 J — be fixed. 

Problem 7.1. Minimize C(p) subject to the constraint f M p dx = 1, where 



(22) 



C(p) = 




-f+-(J 
P 



\dp) 



p dx. 



Remark 7.2. The constraint p > on M does not need to be enforced, since if we find p solving 
Problem 17.11 without this constraint, we may compute u by (fl3)) . Then p satisfies L* u p = SJ = 
by Lemma fe-lO! so by Proposition 13 . 51 and the constraint J M p dx = 1, it follows that p > on M. 

Remark 7.3. Note that by Lemma l3.11[ the contribution of A is determined once we fix J. There- 
fore we may put A = in the current optimization problem. 

Necessary conditions for the solution of Problem 17.11 may be obtained rigourously in a similar 
manner as in Section [5l to obtain the following result. 

Theorem 7.4. Suppose p € C°°(M) minimizes C(p) given by (|22[) under the constraint that 
J M p dx = 1 . Then there exists a p, € K such that 

(23) i A 0-(W + A M )0 = -Mf, 

holds, where <j> = *fp and W = XV + ^\\f\\ 2 - \o~f- 

Remark 7.5. Equation (|23p is known (at least in the one-dimensional case) as Yermakov's equation 

Instead of proving Theorem 17.41 rigourously (which may be done analogously to Section [5]) we 
provide an informal derivation, which we hope provides more insight to the reader. We introduce 
the Lagrangian C : C°°(M) x K -> K by 



JM 



V 



1 

2A 



-f+-(J 
P 



\dp) 



p dx + p 



p dx — 1 



Varying C(p,p) with respect to p in the direction ( g C°°(M) gives 




f + -{J + ¥p) 



+ At + \{-f + ) ) (J+¥p)i-jA J +¥p)( + Yp dC ) 



dC,)pdx 



2p 2 



J I 



rIMpll 5 



4p 



Ap ) - dx, 



where we used the identities 5 J = and S(huj) = —(dh, to) + hSui for h £ C°°(M) and to £ A 1 (M). 
We require that for any direction £ the above expression equals zero, which is the case if and only 
if 



(24) 



Tp A ? + ^ 2 -^ 2 



W + Xp = 0. 



Note that we need to solve this equation for both p and p, in combination with the constraint 
that p > on M and J M p dx = 1 . By substituting <fi — ^fp, equation (I24[) transforms into the 
equation (|2"3"1) . 
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We may then compute the cost corresponding to p - 

2\ 



as 



C( P ) = 



(25) 




p dx 



P_ 

2A 



2A 



2A 



-f + -(j+\dp) 



dx 



M 



4A 



Ap+^-Sf-pp--(f, J) 



2A 



A 



f 'Y P dp 



(J, \dp) dx 



,1/ 



(/, J) 



where we used (f2"4"| in the first equality, and the following observations for the last equality: 



/ Apdx = - (d(l),dp) dx = 0, 

JM J 



p(Sf) dx= I (dp, f) dx, 

M JM 



1 



(J,dp)dx= / (J,d(hxp)} dx = / (bxp)(8J) dx = 0, / p dx = 1. 

We can only influence the first term in (|25p by choosing p or p, so we see that minimizing C 
therefore corresponds to finding the largest value of p such that (j2"4")h or, equivalently, (|23l) . admits 
a solution. 

7.f . Symmetrizable solution time independent Schrodinger equation. In this section 
we consider the special case of the above problem for zero current density, J = 0. By Lemma l5.15l 
this is equivalent to u + / = —\d~$> for some unknown ^ e C°°(M), with p = exp(— 'J'). In 
other words, we demand the net force field (including the control) to be in gradient form, and the 
corresponding diffusion to be symmetrizable; see Section [5.41 
In this case (|23[) transforms into the linear eigenvalue problem, 

(26) iA0 - W<j) = \p<j). 

This is intriguing since this is in fact a time independent Schrodinger equation for the square 
root of a density function, analogous to quantum mechanics; even though our setting is entirely 
classical. 

By (f2"5)) . we are interested in the largest value of p so that (f2T)|) has a solution <f>. The optimal 
control field is then given by 

(27) u=i#-/. 



Remark 7.6. It is straightforward to check that if / — —\dU for some U € C°°(M), then <fi — 
exp(— satisfies (|26|) with V = 0, /x = 0, resulting in it = 0. This corresponds to the intuition 
that, if / is already a gradient, no further control is necessary to obtain a symmetrizable invariant 
measure. 

Remark 7.7. We may also compare the case / = —\dU with the result of Section T5.4I There 
we obtained that, in case A = and / = —^dU, the optimization problem for unconstrained J 
resulted in a symmetrizable solution. In other words, the constraint J — does not need to be 
enforced, and the solution of this section should equal the solution obtained in Proposition l5.16l 
Apparently, with ip as in Proposition l5.161 we have that </> 2 = ip 2 exp(— U). 

8. Discussion 

In this paper we showed how stationary long term average control problems are related to eigen- 
value problems (for the unconstrained problem and the problem constrained to a symmetrizable 
solution, Sections [51 and [7Tj) , elliptic PDEs (for the problem with fixed density, Section [6]) or a 
nonlinear PDE (for the problem with fixed current density, Section [7]). For this we fruitfully used 
the representation of an optimal control field u in terms of the density function p and the current 
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density J. We showed in detail how an infinite dimensional Lagrange multiplier problem may be 
transformed into a PDE (Section[S]). A striking relation between the classical setting and quantum 
mechanics was obtained in Section [7.1l 

The theory on existence of solutions and spectrum of operators is classical and we refer the 
interested reader to e.g. |T0l [19] . Let us again point out the strong connection of our results with 
earlier work of Donsker, Varadhan (1975) [9]; see also Remark 15.121 We will further investigate 
this connection as part of our future research. 

One may ask the question whether we may obtain solutions when we constrain a certain flux 
f M (A, J) dx (see Section |4} to a given value. In this case, one may use A — pA as a Lagrange 
multiplier and use the results of Section[5](for constrained flux) and Section [6] (for constrained flux 
and density) to obtain necessary conditions. As these results did not provide us with profound 
insight, we aim to report on this topic in a subsequent publication after further analysis of the 
problem. 

Appendix A. Derivation of expression for long term average of current density 
In the physics literature [7], the current density is defined formally as 

1 '* 



(28) T t {x) = - / Xl8{X s -x) ds, 

1 Jo 

for x £ M, where S is the Dirac delta function. We will derive an alternative expression for this 
quantity, using the model ^ for the dynamics. Note that (j2"5)l formally defines a vector field that 
acts on functions / £ C°°(M) as 

Jtf(x) = -( X\5{X S - x)d l f{x) ds=- ( X l s S(X s - x)OJ(X s ) ds = - - f 6(X S - x)d l f{X s ) o dX\ 
' JO ' io t Jo 

The 5- function is still problematic. We may however formally compute the L 2 (M, g) inner product 
of the above expression with any h £ C°°(M) with support in a coordinate neighbourhood U 
containing x. This results in 

h{x)J t f(x) dx = I [ h{x) f 5(X S - x)d.J(X s ) odX l s dx = - ( h(X s )dJ(X s ) o dXl 

M t Ju Jo * Jo 



Using the relation Y o dZ = Y dZ + \d\Y, Z] and (0, we compute 

h(x)J t f(x) dx = jj* h(X s )dif(X s ) (t u (X s )ds + a l a (X s ) dBf) «^ W) (X s ) ds 

$yh(x)(dif)(x) (V u {x)j + \g l ^dj (hdtf) (x)\ fi u (dx) (almost surely as t — > oo) 

h(x) I Pu (dJ) (X) - |JL ( di f) 9j (p u ^ftg\9 ij ) } (*) dx, 



using Proposition 13.61 and the law of large numbers for martingales ( |161 Theorem 3.4], or [31 
Section 6.4.1] for a proof). We find that the long term average vector field J has components 



r = 



= Pub l u - \g l ° {d ]Pu ) , 
where the last equality is a result of the identity 



\9\ 

which may be verified by straightforward calculation. Lowering the indices gives the differential 
form 

Ju = -\dp + p(f + «). 
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Note the abuse of notation: both differential form and vector field are denoted by J. The vector 
field denoted by J has no relevance in the remainder of this work. 
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